Activate bzlmod #1912

ylecornec · 2023-06-26T17:35:52Z

Depends on #1911

This PR contains the changes necessary to activate bzlmod in the CI.

A new rules_haskell_nix module is added, which is used to install nix toolchains with bzlmod. This way the main rules_haskell module does not have to depend on rules_nixpkgs (other than as a dev dependency).
- In workspace mode the @rules_haskell//haskell:nixpkgs.bzl file is still available for backward compatibility (it is now a symlink to the file in rules_haskell_nix).
- This @rules_haskell//haskell:nixpkgs.bzl symlink is also used by rules_haskell itselft for testing, but only as a dev dependency.
A local registry is used to access module from rules_nixpkgs until they are available on the central registry, as well as to access local versions of rules_haskell and rules_haskell_nix for testing.
The .bazelrc file is split into .bazelrc.common and .bazelrc.bzlmod. This way the various modules can share only the common part (they need to access the registry using different relative paths).
The new rules_haskell_dependencies extension contains dependencies which are always needed by rules_haskell (as opposed to the ones from non_module_dev_dependencies).
The new rules_haskell_asterius extension generates utility repository for asterius, and provide a custom tag to be able to use webpack from a pre-existing npm setup.

Changes to the runfiles library.

Since the name of repositories in the output base can change, the runfiles library must be able to resolve them.

As explained in this design document, this is done by parsing the _repo_mapping file which is now provided by bazel in the runfiles tree.

Since each repository has its own mapping, there is also some logic to recover the name of the current_repo, which is the repository making use of the runfiles library. A troublesome case is when a haskell library with runfiles is beeing used by a binary of another module. In this case, we cannot rely on the binary path to recover the current_repoof the library.

This is why a new createFromCurrentFile function was added to the runfiles library, to which we can pass the __FILE__ constant in order to recover the name of the module containing the library (similarly to what is mentioned in this section of the design document).

This requires enabling the preprocessor which has some limitation (such as being incompatible with multiline string), so the haskell_runfiles helper rule is provided in @rules_haskell//haskell:defs.bzl, which generates a small wrapper around the main runfiles library (to separate the usage of {-# LANGUAGE CPP #-} from the rest of the code).

Changes to the CI

Both WORKSPACE and bzlmod cases are now tested in the CI. This adds new jobs so the required status checks of github will need to be updated manually again.
On windows, we use a separate remote cache for bzlmod because the dependency analysis of C files is leaking absolute paths which are different in both cases.
The rules_haskell_nix module does not contains tests yet (it is tested by the rules_haskell_tests module in bzlmod mode), but at least buildifier should be added once more files are added to it (most notably the module extension to install nix toolchains).
The tutorial and example folders do not contain MODULE.bazel files yet so they are not tested with bzlmod.

dpulls · 2023-06-27T06:39:10Z

🎉 All dependencies have been resolved !

This commit add a custom registry for bzlmod support. - It is used to access module from rules_nixpkgs until they are available on the central registry - It is used to access local versions of `rules_haskell` and `rules_haskell_nix` modules for testing.

The legacy io_tweag_rules_nixpkgs repository is not a bazel module, so with bzlmod, we use the modules provided with rules_nixpkgs instead.

This module should be used with bzlmod to install nix based toolchains. This way the main rules_haskell module does not depend on rules_nixpkgs.

- bzlmod is not setup yet in examples and tutorial directory

The docs folder has development dependencies that are not available from `rules_haskell_nix` and is not necessary for integration tests.

Various modules will need to reference the local registry with different relative paths.

For non development dependencies of rules_haskell which are not available as modules.

aherrmann

Good work! Thank you!
This was a difficult task,

Just a couple minor comments.

aherrmann · 2023-06-28T13:48:55Z

haskell/nixpkgs.bzl

@@ -1,414 +0,0 @@
-"""Workspace rules (Nixpkgs)"""


Something to watch out for: Symlinks in Zip archives are problematic, so turning this file into a symlink may make .zip bundles of rules_haskell unusable.

.github/workflows/workflow.yaml

rules_haskell_tests/non_module_deps_2.bzl

extensions/rules_haskell_dependencies.bzl

haskell/asterius/extension.bzl

aherrmann · 2023-06-28T15:38:54Z

tools/runfiles/src/Bazel/Runfiles.hs

 import System.Info (os)

+-- | Bazel repository mapping for bzlmod
+-- See https://github.com/bazelbuild/proposals/blob/7c8da4a931d83db5c25abf85f6c486ad22d330e3/designs/2022-07-21-locating-runfiles-with-bzlmod.md
+type RepoMapping = [((String, String), String)]


Perhaps out of scope for this PR, but I wonder if we could overhaul this module to use some better data structures than [a] and String, like Map and perhaps even text (though I guess we want to stick to core package dependencies).

According to this file it seems that text is only a core package since ghc-8.4, but containers is there for all the versions mentioned in the file (which start at ghc-7.10.1).

I updated the library to use Data.Map.Strict in this commit (the lookupDir function makes use of Map.to_list but I think it's fine since we need to iterate over (key,value) pairs in this case).

tools/runfiles/src/Bazel/Runfiles.hs

haskell/defs.bzl

aherrmann · 2023-06-28T15:59:00Z

haskell/defs.bzl

+    native.genrule(
+        name = "{}_genrule".format(module_name),
+        outs = ["{}.hs".format(module_name)],
+        cmd = """
+        cat << EOF >> $@
+{{-# LANGUAGE CPP #-}}
+module {} (create, createFromProgramPath, rlocation) where
+import qualified Bazel.Runfiles as RulesHaskellRunfiles
+create = RulesHaskellRunfiles.createFromCurrentFile __FILE__
+createFromProgramPath = RulesHaskellRunfiles.createFromProgramPathAndCurrentFile (Just __FILE__)
+rlocation = RulesHaskellRunfiles.rlocation
+EOF
+""".format(module_name),


Minor: Maybe this could use Skylib's expand_template or write_file rule?

I updated the PR to use expand_template 👍

See https://github.com/bazelbuild/proposals/blob/main/designs/2022-07-21-locating-runfiles-with-bzlmod.md fixup runfiles

All the documentation is generated from the `rules_haskell` module.

ylecornec · 2023-06-29T06:48:58Z

Thanks! I addressed the comments (except for the symlink one for which I have no good solution at the moment).

The required status checks of github need to be updated manually again to be able to merge, since new jobs were added to test bzlmod.

aherrmann · 2023-06-29T08:25:40Z

The required status checks of github need to be updated manually again to be able to merge, since new jobs were added to test bzlmod.

done.

ylecornec requested a review from avdv as a code owner June 26, 2023 17:35

ylecornec force-pushed the ylecornec/activate_bzlmod branch from 030bbdd to de4fc78 Compare June 26, 2023 17:47

This was referenced Jun 14, 2023

Test bzlmod in CI #1893

Closed

Organize the repository in modules (in particular separate tests and nixpkgs integration) #1895

Closed

Base automatically changed from ylecornec/bazel_6 to master June 27, 2023 06:39

ylecornec added 8 commits June 27, 2023 08:53

Add custom registry

03ef807

This commit add a custom registry for bzlmod support. - It is used to access module from rules_nixpkgs until they are available on the central registry - It is used to access local versions of `rules_haskell` and `rules_haskell_nix` modules for testing.

new rules_nixpkgs_names

8b5fb22

The legacy io_tweag_rules_nixpkgs repository is not a bazel module, so with bzlmod, we use the modules provided with rules_nixpkgs instead.

Add rules_haskell_nix module

be0bf6b

This module should be used with bzlmod to install nix based toolchains. This way the main rules_haskell module does not depend on rules_nixpkgs.

Run bzlmod in CI

c3ee9fc

- bzlmod is not setup yet in examples and tutorial directory

Remove docs from :distribution

b5ed035

The docs folder has development dependencies that are not available from `rules_haskell_nix` and is not necessary for integration tests.

Add dummy stack_asterius repository

51e35d5

Refactor bazelrc files

be56859

Various modules will need to reference the local registry with different relative paths.

Add rules_haskell_dependencies extension

506ae8d

For non development dependencies of rules_haskell which are not available as modules.

ylecornec force-pushed the ylecornec/activate_bzlmod branch from de4fc78 to c555631 Compare June 27, 2023 06:56

ylecornec mentioned this pull request Jun 27, 2023

haskell_toolchains extension #1914

Merged

aherrmann approved these changes Jun 28, 2023

View reviewed changes

Add asterius module extension to declare utility repositories

7a4c0b2

ylecornec force-pushed the ylecornec/activate_bzlmod branch 2 times, most recently from e744032 to 9af561b Compare June 28, 2023 18:18

ylecornec added 2 commits June 28, 2023 20:30

Add MODULE.bazel file for rules_haskell_tests module

0a3d8b4

Fix @rules_haskell_tests//tests/java_classpath with bzlmod

1d61dae

ylecornec force-pushed the ylecornec/activate_bzlmod branch 2 times, most recently from 0c9a5e2 to 8d445fa Compare June 28, 2023 19:18

ylecornec added 6 commits June 28, 2023 22:22

Add bzlmod support to haskell runfiles library

fb7143f

See https://github.com/bazelbuild/proposals/blob/main/designs/2022-07-21-locating-runfiles-with-bzlmod.md fixup runfiles

Add haskell_runfiles rule

9ae2fab

Use Data.Map in the runfiles library

8748511

Add MODULE.bazel file forrules_haskell module.

1a249a3

Test for a haskell library of a separate module accessing runfiles.

4c300bb

Remove stardoc from rules_haskell_tests WORKSPACE file

8877ccc

All the documentation is generated from the `rules_haskell` module.

ylecornec force-pushed the ylecornec/activate_bzlmod branch from 8d445fa to 8877ccc Compare June 28, 2023 20:26

aherrmann added the merge-queue merge on green CI label Jun 29, 2023

Update mergify configuration for bzlmod

ed414bd

ylecornec mentioned this pull request Jun 29, 2023

Bazel ci fixes #1916

Merged

ylecornec merged commit cd23a0c into master Jun 29, 2023

ylecornec deleted the ylecornec/activate_bzlmod branch June 29, 2023 09:51

mergify bot removed the merge-queue merge on green CI label Jun 29, 2023

ylecornec mentioned this pull request Jun 30, 2023

Update rules_haskell runfiles_library #1894

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Activate bzlmod #1912

Activate bzlmod #1912

ylecornec commented Jun 26, 2023 •

edited

Loading

dpulls bot commented Jun 27, 2023

aherrmann left a comment

aherrmann Jun 28, 2023

aherrmann Jun 28, 2023

ylecornec Jun 29, 2023

aherrmann Jun 28, 2023

ylecornec Jun 29, 2023

ylecornec commented Jun 29, 2023

aherrmann commented Jun 29, 2023

Activate bzlmod #1912

Activate bzlmod #1912

Conversation

ylecornec commented Jun 26, 2023 • edited Loading

Changes to the runfiles library.

Changes to the CI

dpulls bot commented Jun 27, 2023

aherrmann left a comment

Choose a reason for hiding this comment

aherrmann Jun 28, 2023

Choose a reason for hiding this comment

aherrmann Jun 28, 2023

Choose a reason for hiding this comment

ylecornec Jun 29, 2023

Choose a reason for hiding this comment

aherrmann Jun 28, 2023

Choose a reason for hiding this comment

ylecornec Jun 29, 2023

Choose a reason for hiding this comment

ylecornec commented Jun 29, 2023

aherrmann commented Jun 29, 2023

ylecornec commented Jun 26, 2023 •

edited

Loading